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ABSTRACT 

A discussion of second language program evaluation is 
intended for staff (teachers, program planners, materials writers, 
program managers) of programs being evaluated, to help them 
understand the process and dynamics of program evaluation. It 
addresses common concerns and fears about evaluation, offers ways in 
which staff can cooperate with evaluators or defend themselves 
against inadequately trained or insensitive evaluators, and provides 
practical suggestions for becoming active participants. First, the 
difference between applied linguistics research (classroom or 
institutional) and program evaluation are enumerated, and a 
distinction is made between advocacy and evaluation. It is suggested 
that a research-like procedure can masquerade as evaluation, and 
eight ways in which program staff can turn this procedure into a 
cooperative process are outlined. All revolve around clear 
communication between staff, source of funding for the study, and 
researcher* (MSE) 
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HOW PROGRAM PERSONNEL CAN HELP MAXIMIZE 
THE UULTTY OF LANGUAGE PROGRAM EVALUATIONS 



Ronald Mackay 
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1 . THE SOURCE OF THE CONTENT OF THIS PAPER 

This paper is based on my own experience as a researcher and as an evaluator in the 
field of language teaching and applied linguistics. It has been derived directly from a 
textbook which I am in the process of preparing for publication. My interest in language 
progr am evaluation has been a natural development arising out of my work in first' teaching, 
then designing oirricula and in materials writing. Over the past decade I have been involved 
in a dozen program evaluations. Some of these have been small-scale, involving individual 
programs, small numbers of teachers and students, offering minimal financial resources for 
the evaluation activity and only one evaluator. Other evaluations have been quite extensive, 
involving entire provincial and territorial school systems, scores of schools, and budgets in 
the hundreds of thousands of dollars. 



2. THE PURPOSE OF THIS PAPER 

The purpose of this paper is not to outline to you what you must learn in order to 
become a program evaluator yourself. There are dozens of "How To books in the field of 
programme evaluation. Some are good, some bad and some merely indifferent. Furthermore, 
not everyone wants, and still fewer have the opportunity to leave teaching or program 
management for the full-time practice of evaluation. On the contrary, this paper is directed 
at those who staff language programs which are likely to be evaluated at some time or other. 
It is directed at teachers, program planners, materials writers and program managers in 
order to help them understand what might happen when their program comes up for 
evaluation. Its purpose is to help them overcome their understandable concerns and fears 
surrounding the evaluation process; to show them ways in which they can cooperate with the 
evaluator and also ways they can defend themselves against inadequately trained or 
insensitive evaluators. Most of all, its purpose is to provide them with specific practical and 
workable suggestions as to how they can Hcome active participants in any evaluation of their 
program. Participation helps to ensure that the results of the evaluation will address their 
concerns, their questions and their interests, reflect their perceptions and contribute to their 
aspirations, and not merely those of some outside evaluator who, Uke the proverbial cat, 
crept in, evaluated, and crept out again. 
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3. SOME PRELIMINARY QUESTIONS 



3.1 Why Do I Consider It Necessary For Program Staff to Learn What I Have, Only 
Semi-Humourously, Called "Self-Defence" ? 

First of all because program evaluation in our field of second language teaching and 
applied linguistia; so far offers little in the way of training for potential evaluators, and has 
articulated no professional standards with which evaluators are uged lo comply. Hence many 
different kinds of person can assume the title of 'evaluator' and many prac*iccs of varying 
kinds and quality arc undertaken in the name of 'evaluation'. 
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Rather than focus upon the ir atter in a negative way, I will briefly outline to you the 
posiitivc steps that are being taken in Canada to deal with the growing interest in evaluation. 
Six years ago the Canadian Evaluation Society was established with these objectives : 

* To provide a forum whereby program managers, administrators, policy-makers, 
practitioners teachers and students can discuss the theory and practice of evaluation in 
Canada; 

* To promote the high quality of evaluation of public and private programs throughout the 
country; 

* To develop theories, standards, and practices of evaluation; 

* To promote training programs in planning the design, strategy, methods, analysis and 
application of results for all types of evaluation; 

* To provide a forum of exchange on policies, practices, applications and sources of funding 
for evaluation. 

The C.E,S, draws its membership from a diverse group of professionals in the public 
private and academic communities who share a common interest in evaluation They 
represent disciplines ranging from psychology sociology, social work, economics, health 
sciences, administration, political science, and policy sciences to accounting, engineering and 
urban and re^onal planning. The C.E.S. holds a national convention, publishes a journal and 
a newsletter, and is organized around regional groups which arrange meetings and 
professional development sessions for their members. 

This is one way of tackling the growth of and interest in evaluation - namely to encourage 
the professional development of those who practice evaluation. This paper is another, 
complimentary way of tackling the issue - namely it encourages the interest of program staff 
in becoming more informed about evaluation and its implications so that they can participate 
with the evaluator in the evaluation process and so that evaluation is not something that is 
done to them, but for them and with their consent, cooperation, and understanding. 



3«2 "What Is the Difference Between Research and Evaluation ?" 

Applied linguistics research and second language program evaluation are for the most 
part two different and distinct activities.They can be distinguished from eachother on a 
number of counts. Some of the differences listed below (see also, Popham 1975; Patton 1986; 
Worthen and Sanders 198" ) overlap somewhat, and the list is not exhaustive.: 



THE IDENTIFIABLITY OF THE CLIENTS 

Research is not usually carried out for an identifiable client. Research tends to be 
funded by governmental or philanthropic organizations, but it is not carried out for their 
benefit. An evaluation is funded by a sponsor who may be equally as faceless as a 
government department or a philanthropic organization but is carried out for a specific client 
or group of clients, one of whom may or may not be the funder. These clients are identifiable 
as individual people with a particular identifiable interest or stake (as it is called) in the 
results of the evaluation. 
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THE REIAUVE INDEPENDENCE OF THE INQUIRER 



Researchers in effect work for themselves in the sense that they decide what question 
to research, what focus they will adopt, v^at variables they v/iHl manipulate, which they will 
hold constant and which they will ignore altogether. Evaluators on the other hand answer 
questions posed by stakeholders, focus on the latters' consems and Interests and examine 
issues selected by those funding, mana^g, teaching in or in some other way affected by the 
program. Researchers normally initiate their own research projects based on their own 
personal curiosity and interests. They decide what they want to investigate and then seek 
the funds from an appropriately sympathetic organization to carry out the investigation. 
Evaluators respond to the information needs of others, normally stakeholders in a 
particular program. 



THE USEFULNESS OF THE INQUIRY 

Research is not motivated by immediate utility, whereas evaluation is. The results of 
an evaluation will be used by a client or by different groups of clients ( called the 
stakeholders) to answer their questions, confirm their suspicions, and inform their decisions 
concerning one or more aspects of the program that they design, manage, operate, teach in, 
learn in or are in some way or another affected bj- . 



THE MOTIVATION OF THE INQUIRER 

The researcher is motivated by a desire to expand the frontiers of human knowledge; 
the evaluator is motivated by the desire to provide illuminating answers to specific 
questions posed by spC'dTic stalkeholders about a specific program in a particular context. 



THE OBJECT OF THE INQUIRY 

Research strives to further knowledge, serve truth and broaden our understanding of 
the world and the types of phenomena (e.g. types of programs ) and so tries to focus on the 
typical obtained by means of random sampling (or some similarly scientific strategy of 
selection) vrithin the populations they wish to generaluize to. Evaluation, on the other hand, 
is concerned with the interests and concerns of specific people involved with specific 
programs and so focuses on the individual project within its own unique context. 



GENERALIZABILITY 

The researcher seeks to uncover the general laws of nature usually by means of 
establishing significant relationships between variables. Evlauators focus on specific 
contextrs, particular programs v^th all their individual constraints and idiosyncracies. 
Whereas the researcher wants to make strong and broad generalizations the evaluator wants 
to enlighten individual stakeholders involved with individual programs. Researchers seek to 
uncover general laws which govern human learning. Evaluators seek to describe a specific 
program identifiable in time and place, in an illuminating way. and, in the process, to 
provide stakeholders with a better understanding of its strengths and weaknesses. 
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THE ROLE OF EXPIANANTION 

Researchers seek to explain the stable relationships between variables ideally in a 
cause and effect relationship. Evaluatosrs seek to provide credible exlanations to 
stakeholders for what is occurring in their program so that specific adjustments might be 
made to improve it. 



RELEVANCE OF TIME" 

More often than not, the researcher works to self-imposed deadlines. These 
deadlines derive from factors extrinsic to the research - dates related to the university 
calendar, dates related to the annual calendar for funding by grant agencies.The value of 
research results are unlikely to be wiped out if the research project is completed a little 
behind schedule. Evaluators work to deadlines intrinsic to the project that they are 
evaluating. Stakeholders require information by certain times so that effective decisions 
can be made. Information received from a tardy evaluator one day after a major project 
planning meeting of the stakeholders may render that information generated by the 
evaluation totally useless. 



DISCIPLINE LOYALTY 

Research normally is undertaken within the well-defmed ( but often arbitrary) 
paremeters of speciHc disciplines. Evaluations seldom demand loyalty to one particular 
discipline and often involve the evaluation team working cooperatively across specialization 
boundaries. 



SOME REASONS FOR THE CONFUSION BETWEEN 
EVALUATION AND RESEARCH 

First, training courses for second language program evaluators are uncommon but 
instruction in research methods based on the experimental paradigm are not. Thus even 
the m ost willing of evaluators may be the victims of one particular, and very limited, frame 
of reference. Second, when research funds are tight, enterprizing researchers may seek 
evaluation contracts to continue their research work, consciously or unconsciously defining 
the focus of the evaluation to coincided with their own interests. Third, some sponsors of 
evaluations may make money available for an evaluation \^thout providing adequate terms 
of reference for the evaluation, thus leaving the researcher to determine the focus and 
direction of the activity him or herself. 



33 What is tbi; difference betweenm functioning as an evaluator and functioning 
as an advocate? 

Evaluation and advocacy are legitimate activities and both may be conducted in a 
professional manner. The function of an advocate however is to present a program in the 
best possible light in order to influence the decisions made by a major information user - 
usually the funder - in a particular way - usually to continue funding or to incresc the funds 
for the program in question. The function of an evaluator cannot be so one-sided. An 
evaluator who functions as an advocate will lose his/her credibility as an independent judge. 
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Some professionals, those who are committed to one of the many forms of sodal reform, for 
example, may believe that their moral duty is to advocate. There is nothing wrong with that 
so long as they are up-front about their position and do not present themselves or their 
activities as evaluation. 



4, A RESEARCK-LIKE PROCEDURE WHICH MASQUERADES 
AS EVALUATION 

When research masquerades as evaluation, it tends to follow a particular process (Fig. 

1). 



FIGURE 1 

RESFARCH-UKE EVALUATXW PflOCEDURE 

FUNDER iNmATES 
EVALUATOR DEaOES FOCUS 
EVALUATDR DESiONS EVALUATION 
EVALUATOR GATHERS DATA 
EVALUATOR ANALYSES DATA 
EVALUATOR INTERPRETS DATA 
EVALUATOR MAKES RECOMMENDATIONS 
EVALUATOR WHTTES REPORT 
EVALUATOR DEPOSITS REPORT 
FUNDER RECEIVES REPORT 



6 



64 



R£SEARCH*LIKE EVALUATION PROCEDURE 

This not uncommon research-like procedure which is sometimes passed of by its 
practitioners (purposely or unconsciously) as evaluation has little time for the stakeholder's 
concerns and jillows little opportunity for input from the program staff except at those pomts 
where the researcher imilaterally decides that it is necessary for his or her purposes to obtain 
information, usuaUy of a predetermined kind« from them. 



5. How can program staff make a difference ? 

Thus far we have painted a pretty dismal picture - research which masquerades as 
evaluation, evaluators who are insensitive to the stakeholders, advocates in disguise, 
funders who disburse funds without due care and attention to what will be given in 

retiun How can we as program staff change the picture ? How can we as program staff 

ensure that the researcher does not run away with the evaluation funds for his or her own 
purposes which may have marginal or even no relevance to the purposes of the principal 
infotrmation users and the other stakeholders ? How can we as program staff get what we 
want and need out of a program evaluation ? 

The remainder of this paper will be devoted to suggest! ig ways that program staff can 
go about gaining a foothold in the evaluation activities and by so doing, become 
instrumental modifying, to a greater or lesser extent, the research-like process. I will deal 
vnth ways staff can mtervene in two parts. The first part will deal with early and successful 
intervention and will show the cooperative evlauation process which results. The second part 
will deal with later and less successful intervention, and will illustrate what I call the utility 
enhanced evaluation process, in this paper I will not deal with a third part which shows 
program staff what action might be appropriate when it becomes clear that their attempts at 
gaining access to the evaluation process are doomed to failure. You will have to buy the book 
to find that out ! 



5*1 Turning the Research-Like Procedure Into A Cooperative Process 

I vAU take the research-like process discussed above and suggest how, early on in that 
process, the stakeholders can gain a foothold and so help ensure that the evaluation results 
in a report that can be used by them. 

Let's start at the very beginning at the point where the Funder Initiates an 
Evaluation, in Fig. 1. 

For any program, somebody holds the purse strings to the fund from which any 
evaluation to be undertaken \^11 be financed. Sometimes the holder of the purse strings is 
not a major information user or even not a stakeholder at all. This is the case in many 
government sponsored programs. In yet other cases, the funder is a stakeholder and a 
potential information user and may know virtually nothting about evaluation and so hires an 
evaluator to take control. Sometimes the funder simply assumes that he or she is the only 
person interested in the evaluation and does not even consider that others may have a 
substantial stake in any inqiry and may have questions which they would like answered in 
order to make program -related decisions. 
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This stage, because it is the pomt at which the evaluation is initiated, is the most crucial stage 
at which program staff can help to influence the entire subsequent evaluation process.The 
very first task of the enterprising and concerned stakeholder is to find answers to the 
following questions : 

#1 FIND ANSWERS TO THE FOLLOWING QUESTIONS : 

Is it planned that the program will be evaluated ? 
If so, when is it scheduled to happen ? 
Where will the funds come from ? 
How much will they amount to ? 

Are there any constraints upon the type of evaluation that must be carried out ? 

For example does the plan specify whether it will be an internal or an external 
evaluation ? Will it be formative or summative ? Will it be undertaken for a predetermined 
purpose, e,g. to determine whether the program should be expaned to other schools, 
provinces or states ? or whether there are more cost-effective ways of achieving the same 
results ? or whether the program is achieving the purposes for which it was originally 
mounted ? 

When you have found accurate answers to these questions, and you have established 
that there is the potential for your mterests in the program to be addressed,ask yourself, " 
Whet do I need to know about this program ?" and "How will I use that information ?" 
Notice that the operative words here are 'need' and 'use*. It is as a potential user of 
evaluative information that you have the right to intervene or to gain access to the evaluation 
process, not simply as a curious by-stander. For example, as a teacher you might need to 
know if the strategy of selective error-correction that you are employing is having immediate 
and or lasting effects on student production so that you can continue, modify or discontmue 
the practice; as a materials witer or as a head teacher, you might need to know if your 
materials being used in the way and under the conditions they were intended to be used, 
etc.Once you have identified the information you need : 



#2 LET THE FUNDER KNOW THAT YOU HAVE SPECIFIC QUESTIONS YOU 
REQUIRE ANSWERS TO, OR ISSUES THAT YOU WISH TO HAVE 
ADDRESSED. 

In other words, let the funder know that you are a stakeholder, and precisely what 
your stake is in the evaluation This may come as a surprise to a funder who has not given 
consideration to any program staff as being stakeholders. 



#3 TELL THE FUNDER HOW YOU WOULD MAKE USE OF THE INFORMATION 
GENERATED BY THE EVALUATIOlSi 

In other words, let the funder know that you have given serious consideration to your 
stake in the evaluation and that addressing your concerns or questions is important for the 
future of the program. 

If, at this stage, the funder embraces you as a full partner in the evaluation process, 
most of your serious problems are over. You have switched right over from the danger that 
a research-like procedure will be employed to the liklihood that a cooperative procedure 
(Fig. 2) will be adopted. 
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Figure 2 

CQQPERATTVF EVALLiATmN PRQCFnURp 



FUNDER INITIATES 

CUEKT AND STAKEHOLDERS IDENTIFY PIITS 

CUENT AND Fiifs IDENTIFY Questions and Concerns 

CUEm AND PIU'S IDENTIFY USE 

CLIENT AND PIU*S DRAW UP Request for Proposals 

CONTRACTORS SUBMIT TENDERS 

CUENT (ANDPIU^) SELECT WINNER 

CUENT DRAWS UP CONTRACT 

CUENT AWARDS CONTRACT 

EVALUATDR CONDUCTS EVAU1ATK)N ACCORDINQ TO CONTRACT 

But do not relax your attention or your guard! There is a great deal of solid work to 
do, however. Once the £under has agreed that the concerns other than simply those of the 
funder should be addressed, the matter of who the other principal information users (PIU's) 
are and the issue of who addresses their questions (i.e. who the evaluator is), is no longer a 
trivial question. Any evaluator hired, has to be able to address the stakeholders* concerns 
and questions, and tackle the task of answering them in a manner appropriate to your 
requirements, within the time constraints imposed by the need for subsequent action, and 
within the financial constraints imposed by the evaluation budget. No small task to 
administer ! 

The formal way of finding an evaluator to do all this, is by listing the requirements of 
the stakeholders into a document called a Request for Proposals usually referred to as the 
RFP. The minimum contents of the RFP are : 

1. a description of the context of the program 

2. a clear statement of the purposes of the evaluation 
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3. a list of the stakeholders in the evaluation 

4. a list of the principal information users 

5. the time frame wthin which the evaluation must be completed 

6. the financial resources available for the evaluation 

7. the fonn(s) in which the final report(s) is/are to be presented. 

Potential evaluators arc then invited to respond, at their own expense, to the RFP in a 
competitive process known as 'tendering'. The tender is usually advertised in periodicals read 
by the kind of professional you are seeking to attract. Alternatively, the stakeholders might 
suggest three or four names of individuals or groups known to have the skills required to 
address the contents of the RFP. 

Adequate responses to the RFP should : 

1. be clear, concise and jargon-free 

2. ■ clearly itemize all the costs involved in the proposed evaluation 

3. show the qualifications and experience of the evaluator(s) to undertake the 
evaluation 

4. clearly specify the tasks that would be undertaken ( and their sequence) in 
order to complete the evaluation 

5. give time-lines for each task 

6. describe the methodology that would be used and how it would achieve the 
purposes of the evaluation 

7. demonstrate convincingly that all the questions in the RFP would be answered 
appropriately 

You and the funder, in your new cooperative relationship, can examine the potential 
evaluators* responses to the RFP and choose the one that meets your needs most closely. 
Once an evaluator has been chosen, the funder enters into a formal contractual agreement 
with him/her. The contract will, as a minimum, specify : 

1. what the duties of the evaluator are 

2. what the responsibilities of the funder are 

3. what the total budget will be 

4. how and when the evaluator will be paid 

5. how the evaluation work plan can be amended (if at all) 

6. who the person acting as liason between the evaluator and the funder is 
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5^ Turning the Research-Iike Procedure into a UUiity-Enhanced Procedure 

Let us imagine, for the purposes of the rest of the paper, that the funder has retained 
the sendees of an evaluator without consulting the stakeholders. What then ? How can you, 
the stakeholders gain a foothold in the evaluation process ? For the purposes of this paper, I 
will not deal with every step of attemptmg to convert the Research-like process into a utility- 
enhanced process, nor will I give a large number of suggestions or examples. I am sufficiently 
realistic to realize that if I were to be exhaustive, you would go to sleep, and sufficiently 
mercenary to fear that you might not buy my book when it appears !! So I will be relatively 
sucdnt, and selective. 

So let us start one step on in the research-like process where The Funder Retains 
the Services of an Evaluator (Fig 1.) 

At this pomt it is mvaluable to have the stakeholders participate as a group, identify 
their needs and concerns, become familiar with the needs and concerns of their fellows, and 
agree, if at all possible, upon a hierarchically ordered of questions that they would like to 
see addressed. So, 



#4 MEET WITH THE OTHER STAKEHOLDERS AND HAVE THEM IDENTIFY 
WHAT INFORMATION THEY REQUIRE FROM THE EVALUATION AND HOW THEY 
WILL USE THAT INFORMATION 

It must be borne in mind that the stakeholders must be able to show how the 
answers to their questions will be used to make decisions about the program. Questions 
asked out of curiosity, no matter how mteresting they may be, or information which cannot 
be used are not legitimate concerns to present to the evaluator. For example, the project 
manager might like to know whether the teachers with more experience, or the teachers with 
more recent qualifications are having the greater success in implementing a new set of 
materials involving non-traditional classroom interaction. However, if because of union 
agreements there is no possibility of removing the less successful group or even of msistmg 
on obligatory in-service training for them, the infomation, interesting though it may be, is not 
capable of being acted upn and so the question is not a legitimate one for the evaluation to 
address. 



#5 TALK TO THE EVALUATOR 

Program staff may feel that it is somehow "not fitting" to talk to the evlaluator about 
the program and about his/her plans for evaluating it; or they may feel discouraged by the 
evaluator appearing to weant to hold them at arms length from his/her activities. However, 
not only is it fitting to talk to the evaluator, it is essential to talk to him or her. 



#6 ASK THE EVALUATOR TO EXPLAIN TO YOU AND THE OTHER 
STAKEHOLDERS HIS/HER APPROACH TO AND VIEWS ON 'GOOD 
EVALUATION' 

A good evaluator can explain where he/she is coming from and what he/she is doing, 
in the language of the stakeholders. If the evaluator cannot, then the evaluator is not worth 
hiring. An evaluator who has one and only one perspective on evaluation and tries to impose 
that upon the program and the program staff is not worth hiring. But (for the purposes of 
this paper at least) the evaluator has already been retained ! So : 
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#7 LET THE EVALUATOR KNOW THAT YOU HAVE SPECIFIC QUESTIONS TO 
WHICH YOU REQUIRE ANSWERS OR CONCERNS WHICH YOU WOULD 
LIKE TO SEE ADDRESSED* 



#8 LET THE EVALUATOR KNOW HOW YOU WOULD MAKE USE OF THE 
INFORMATION GENERATED BY THE EVALUATION IF IT V^HERE TO 
ADDRESS THESE QUESTIONS AND CONCERNS 



A good evaluator will be interested in the perspectives of the stakeholders - their 
views on v^liat they believe evaluation to be, how it is carried out, how they conceive of their 
program, what their concerns are and what they believe their infomation needs to be and 
why.If you let the evaluator know that you are counting on the results of the evaluation to 
obtain specific information which is otherwise unavailable in order to take action in some 
area of the program it is likely that he/she will not ignore your request,No evaluator worth 
his or her salt will choose to ignore questions asked by stakeholders who are also principal 
information users in favour of self-generated questions. A good evaluator will help the 
stakeholders to discover the right relationship between his/her expertise and their own 
contribution. A good evaluator will appreciate the feeling of self worth that can be fostered 
by encouraging the stakeholders to participate in the evaluation. Conversely, he/she will 
understand the feelings of suspicion, hostility, and disempowerment which will be 
engendered in the stakeholders if they are held at arms length, their concerns spu/ned, their 
questions ignored and their participation rejected.One of the principal advantages to be 
gained from an evaluation is the enlightenment gained by the stakeholders from an 
understanding of and participation in the process itself. 

If, at this point, the evaluator demonstrates a willingness to work with the stakeholder 
group as partners in the evaluation, you have achieved an early entry into the value-enhanced 
process* It is clearly by no means an identical process to that described as cooperative above 
(Fig. 2), because you are stuck with an evaluator you did not select on the basis of a request 
for proposals and this evaluator despite his or her current willingness, may never have 
worked cooperatively with a stakeholder group before, or he/she may have a very limited 
perceptual framework for conducting evaluations, or he/she may at any time feel threatened 
by the new relationship and wish to end it. Time prevents me from offering you suggestions 
as to how to handle the various problems that can arise from this point on. Time also 
prevents me from offering you suggestions as to what you can do if your appeals to the 
evaluator at this step are unsuccessful. Suffice it to say that what we have called the value 
enhanced model of prograipmc evaluation is limited in its usefulness to programme staff in 
proportion to the pomt at which you gain entry to the process. The earlier you gain entry, the 
greater the opportunity for you to influence the focus of the evaluation, the type of data 
collected and to participate m the interpretation phase and therefore the more potential for 
utility the results of the evaluation wiD have. 



6, CONCLUSION 

You will, however, no doubt appreciate that if you have failed at this point, it may be 
even more difficult (but certainly not impossible) to gain access to the process at one of the 
later stages. Nevertheless, the potential for increasing the utility of an evaluation diminishes 
in direct relation to the tardiness of the phase at which you get entry. It is likely that there is 
a pomt of duninishing returns and a whole new strategy involving a different type of thinking 
is required if you should be so unfortunate as to reach that point without having persuaded 
the evaluator to allow you into the process. If you are unsuccessful in gaining access to the 
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evaluation process at all, then you simply end up with a researcher-directed study 
masquerading as evaluation! Such a study may be instrumental in furthering the researchers' 
interests by providing him/her with a publication or even a higher degree, but it will be 
unlikely to be of use or even of much interest to programme staff concerned about improving 
their activities or finding answers to questions which they require to make informed 
decisions. 

I hope, however, that I have been successful not only m persuadmg you as program 
personnel that it is worth gaining access to the evaluation process, but also m showing you some 
practical ways by means of which you might successfuly become a respected partner m the 
evaluation of your own program. 
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